Skip to content

ADFA-4739: Pipeline for producing template-based Kotlin documentation - #21

Open
alexmmiller wants to merge 16 commits into
mainfrom
fix/ADFA-4737
Open

ADFA-4739: Pipeline for producing template-based Kotlin documentation#21
alexmmiller wants to merge 16 commits into
mainfrom
fix/ADFA-4737

Conversation

@alexmmiller

@alexmmiller alexmmiller commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

An end-to-end pipeline for producing templated Kotlin documentation.

Changes

Kotlin website docs

  • Adds Python scripts used to convert kotlin-web-site/docs (Writerside Markdown) into JSON, build nav, optimize media, and insert everything into documentation.db, into ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/. Adds a README documenting each script's purpose, required inputs (kr.tree, webHelpImages.zip, config JSON), and end-to-end usage — including populate_db.py --blacklisted-element-titles, the mechanism this ticket needs for pruning docs sections (e.g. Kotlin/Wasm) we don't want to ship.

Kotlin-stdlib docs

  • Adds a sync_kdoc_json_to_db.py script into a new directory scripts/sync_kotlin_stdlib_docs/, for syncing generated kotlin-stdlib docs against the database.

Templated docs pipeline

  • Add ProcessDocs/ProcessKotlinDocs/run_e2e_pipeline_test.sh which builds the kotlin-stdlib and Kotlin website JSON documentation and inserts it into CoGo's documentation database.

Test plan for Kotlin website docs:

  • Reviewer confirms the new scripts match their originals in /home/alex/ADFA/KotlinWebsiteProcess/ and Dokka-plugin-kdoc2json/scripts/ (no edits were made during the copy)
  • README instructions are accurate for running populate_db.py with --blacklisted-element-titles against a scratch database

Note on KDoc-to-JSON Dokka plugin version

We currently include a stale copy of the KDoc-to-JSON Dokka plugin while that work is still being done this PR. Once those changes are merged into main we'll merge them into this branch.

alexmmiller and others added 13 commits July 1, 2026 15:42
configJson was a dedicated Json instance used only for the manual
config-decode fallback path. Inlining it removes the standalone
property while keeping the same ignoreUnknownKeys tolerance for
extra keys in user-authored plugin config.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- scripts/verify_package_index.py: confirms every object listed in a
  package's index.json actually has documented content on the page its
  url points to (matching by dri), not just that the file exists.
- scripts/kotlin/test_kotlin_stdlib.sh: pure-bash check that the
  default HTML build and the kdoc-to-json JSON build have a one-to-one
  set of pages (extension-swap aware, in both directions).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…, generate and add templates to display it to database, optimize and insert Kotlin website media. Script to sync current kotlin-stdlib documentation against a newly-generated documentation set (for now, used to do pruning for ADFA-4737 https://appdevforall.atlassian.net/browse/ADFA-4737)
@davidschachterADFA

Copy link
Copy Markdown
Collaborator

Claude says:

Issues found

  1. README is out of sync with what's actually in the PR.
    The README says (lines 32–35):

per ADFA-4737, only the .py files from the original working directory were copied here. templates/ and assets/ ... must be placed alongside these scripts before running populate_db.py.

But the PR itself does include templates/{page.peb,nav.peb,nav.html}, assets/{docs.css,tabs.js,sidebar.js}, and config.json (added by the "Config/assets for pipeline test" commit). A reviewer or new user following the README literally would go hunting for files that are already sitting right there. Worth updating that note (or clarifying it only applied to the original copy-over before the pipeline-test commit added them).

  1. sync_kdoc_json_to_db.py defaults --db to a personal path.

python
parser.add_argument("--db", default="/home/alex/documentation.db", ...)

This is clearly a leftover from the author's local machine. It's harmless in that a wrong path just fails the os.path.isfile check, but it shouldn't ship as the default in a shared repo — either make it required (no default) or default to documentation.db in cwd, matching populate_db.py's convention.

  1. Possible chain-rename bug in insert_optimized_media.py's rewrite_pages.
    rewrite_pages iterates rename_map.items() and does sequential str.replace calls on the same new_text buffer:

python
for old_name, new_name in rename_map.items():
...
new_text = new_text.replace(old_ref, new_ref)

If one rename's new_name happens to equal another rename's old_name (e.g. foo.png → foo.webp from one optimization, and some unrelated foo.webp → foo-2.webp renamed elsewhere), the second substitution could re-match text that the first substitution just wrote, double-rewriting a reference that was never actually pointing at the second file. It's a narrow edge case (requires two renames whose old/new basenames chain), but since build_rename_map only guards against the same old name mapping to two different new names, not this chain scenario, it's worth at least a comment acknowledging the assumption (or building a single combined regex/dict-based single-pass substitution instead of N sequential replaces).

  1. Minor: sync_kdoc_json_to_db.py doesn't explicitly BEGIN the way the other two DB-writing scripts do (conn.execute("BEGIN")). Python's sqlite3 module will still implicitly open a transaction before the first DML statement with the default isolation level, so this isn't a correctness bug, but it's an inconsistency with the explicit-transaction style used elsewhere in the same PR — could confuse a future reader into thinking it's unguarded.

- README: templates/assets are already included in the repo, not left
  for the user to place separately.
- sync_kdoc_json_to_db.py: default --db to documentation.db in cwd
  instead of a personal /home/alex path; add explicit BEGIN to match
  sibling scripts' transaction style.
- insert_optimized_media.py: rewrite_pages now substitutes renamed image
  references in a single regex pass over each row's original text,
  instead of sequential str.replace calls on a mutating buffer, which
  could double-rewrite a reference if one rename's new name chained into
  another rename's old name.
- run_e2e_pipeline_test.sh: add a verification step that reuses
  populate_db.py's own pruning logic to confirm blacklisted topic pages
  are actually absent from the database, replacing a weaker proxy check
  that only tested for k/html/wasm% paths.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@alexmmiller

alexmmiller commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

e2e_run_output.log
I attached the log from running the updated pipeline in the most recent commit.

Summary:

== Summary ==
Database: /Users/alex/test_db_assets/documentation.test.db
k/html/* rows: 502
k/html/images/* rows: 235
k/html/images/.webp rows: 161
k/kotlin-stdlib/
rows: 3132
k/kotlin-reflect/* rows: 65
k/kotlin-test/* rows: 41

== Blacklist pruning verification ==
Blacklisted toc-element path(s) checked: 3
Development > Web development: matched
Interoperability > JavaScript interop: matched
Interoperability > Swift/Objective-C and C interop: matched
Topic page(s) expected removed: 38
PASS: all 38 blacklisted topic page(s) confirmed absent from /Users/alex/test_db_assets/documentation.test.db.

Done. Backups (populate_db.py, insert_optimized_media.py, and sync_kdoc_json_to_db.py
each make their own) live alongside /Users/alex/test_db_assets/documentation.test.db as documentation.test.db.backup-* and
documentation.test.db.bak.*
The real database at /Users/alex/test_db_assets/documentation.db was never opened for writing.
EXIT_CODE=0

@alexmmiller alexmmiller changed the title ADFA-4737: Add Kotlin website doc processing scripts and stdlib doc sync script ADFA-4739: Pipeline for producing template-based Kotlin documentation Jul 23, 2026
alexmmiller and others added 2 commits July 27, 2026 16:04
Brings in the updated KDoc-to-JSON Dokka plugin (sourceSet whitelisting,
package-index/structure verification scripts) so this branch has the
current plugin used to generate the kotlin-stdlib docs consumed by
sync_kdoc_json_to_db.py.
- run_e2e_pipeline_test.sh: replace the manually-supplied STDLIB_ALL_LIBS
  path with STDLIB_DOCS_DIR (a kotlin repo's libraries/tools/kotlin-stdlib-docs
  checkout). New Step 4/5 derives the repo root and calls
  build-stdlib-json-docs.sh to freshly build/publish the kdoc-to-json plugin
  and generate kotlin-stdlib/-reflect/-test JSON docs (common+jvm source
  sets only, per the existing plugin config) before Step 5/5 syncs them into
  the database.
- Add build-stdlib-json-docs.sh: builds a fresh copy of the plugin, swaps in
  the JSON-plugin-enabled build.gradle.kts for the duration of the build
  (restored on exit via trap), and prints only the resulting all-libs path
  to stdout so it composes as STDLIB_ALL_LIBS="$(build-stdlib-json-docs.sh
  <kotlin-repo-root>)" - every gradlew invocation inside it is redirected to
  stderr so that capture isn't polluted by build console output.
- build.gradle.kts: re-add the Dokka dev-snapshot Maven repo (same
  dokka_repository property/default kotlin-stdlib-docs' own
  settings.gradle.kts uses) - kotlin-stdlib-docs' own
  dokka-samples-transformer-plugin subproject needs a Dokka dev build that
  isn't on Maven Central, and the allprojects{} repositories block here had
  removed access to it for every subproject.

Verified end-to-end against a real kotlin checkout: fresh plugin build,
JSON generation (common+jvm only), and sync into a scratch documentation.db
all succeed, with blacklist pruning verification passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants